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DETAILED ACTION 

This action is responsive to communications: RCEX filed 01 May 2008. 
Claims 1, 2, 5-15, 23, 24 and 29-31 are pending in this case. Claim 1 is an 
independent claim. 

Applicant's Response 

In Applicant's response dated 01 May 2008, Applicant amended claims 1, 5, 6, 9, 
12 and 15; canceled claims 16, 17, 20-22 and 25-28; added new claims 29-31; argued 
against all objections and rejection previously set forth in previous Office Action. 

Continued Examination Under 37 CFR 1.114 

A request for continued examination under 37 CFR 1.114, including the fee set 
forth in 37 CFR 1 .17(e), was filed in this application after allowance or after an Office 
action under Ex Parte Quayle, 25 USPQ 74, 453 O.G. 213 (Comm'r Pat. 1935). Since 
this application is eligible for continued examination under 37 CFR 1.114, and the fee 
set forth in 37 CFR 1.17(e) has been timely paid, prosecution in this application has 
been reopened pursuant to 37 CFR 1.114. Applicant's submission filed on 01 May 2008 
has been entered. 
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Claim Objections 

Claim 13 is objected to because of the following informalities: The claims recites 
"The method of claim 1, wherein the determination of the propagation of annotation is 

made "Claim 1 does not disclose a step of determination of the propagation of 

annotation. Claim 1 simply discloses "propagating said input annotation " 

Appropriate correction is required. 

Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

Claims 1, 6, 10 and 23 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Stubler et al. (Patent No.: US 6,804,684, B2; Filed May 7, 2001) 
(hereinafter "Stubler") in view of LeBrun et al. (Patent No.: 6,043,819; Filed Jul. 01, 
1997) (hereinafter "LeBrun), further in view of Neal et al. (Patent No.: US 6,697,799 
B1; Effective Filling Date: Sep. 10, 1999) (hereinafter "Neal"). 

In regards to independent claim 1 Stubler discloses method for generating 
persistent annotations of multimedia content, comprising one or more repetitions of the 
following steps: 
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actively selecting examples of multimedia content to be annotated by a user (fig. 
2, fig. 6-7, col. 3 line 46 - col. 4 line 12, col. 8 lines 18-23, and col. 9 line 65 - col. 10 
line 18; col. 11, lines 3-35; Stubler discloses unlabeled image regions being presented 
to the user so that the user can apply a caption or label to all of the regions 
simultaneously. Stubler also disclose automatically generating one or more captions or 
labels for the acquired image automatically without any user intervention.). 

accepting input annotations from said user for said selected examples (fig. 2, col. 
3 line 46 -col. 4 line 12, and col. 8 lines 18-55; Stubler discloses unlabeled image 
regions being presented to the user so that the user can apply a caption or label to all of 
the regions simultaneously.). 

propagating said input annotations to other instances of multimedia content (fig. 
2; col. 3 line 46 - col. 4 line 12; col. 8 lines 18-55; col. 2 line 59 - col. 3 line 10; Stubler 
discloses unlabeled image regions being presented to the user so that the user can 
apply a caption or label to all of the regions simultaneously.). 

storing said input annotations and said propagated annotations (col. 8 lines 18- 

55). 

Stubler does not expressly disclose actively selecting examples of multimedia 
content to be annotated by a user, wherein the examples of multimedia content are 
selected based on at least one criterion for achieving a maximal disambiguation result 
such that only those examples which are most ambiguous are selected, the at least one 
criterion including a quantitative measure of confidence in a label. 
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However LeBrun teaches actively selecting examples of multimedia content to be 
annotated by a user, wherein the examples of multimedia content are selected based 
on at least one criterion for achieving a maximal disambiguation result such that only 
those examples which are most ambiguous are selected (col. 3, lines 49-51 ; col. 10, 
lines 4-8; col. 18, lines 16-20; col. 21, lines 20-24; LeBrun teaches a database of 
graphic document images which are automatically identified (annotated) by an image 
character reader. Images not automatically identified (annotated) by the image 
character reader are queued for manual identification/classification by human operators 
looking sequentially at a screen or queue of images and keying in the identification.). 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Stubler with LeBrun for the benefit of speeding up the 
process of document (multimedia annotations) processing so that a higher volume of 
transactions can be processed, and reduce the number of errors which are considered 
to be inherent in a document processing (multimedia annotations) operation (col. 3, 
lines 14-23). 

Neal teaches the at least one criterion including a quantitative measure of 
confidence in a label (col. 1 1 , lines 1-61 ; Neal teaches a classification confidence score 
which determines the level of confidence in which a category is likely to be correct 
during classification. If the item has a high confidence, then it can be classified directly. 
If the confidence level is low, then the results can be sent to the user interface for 
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review and selection by the operator. Neal further teaches comparing the confidence 
score to a threshold to determine whether or not to automatically classify an item. Using 
the broadest reasonable interpretation, the Examiner has determined that the 
"confidence score" being compared to a threshold value as taught by Neal is analogous 
to at least one criterion including a quantitative measure of confidence in a label of the 
current invention.). 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Stubler and LeBrun with Neal for the benefit of 
comparing confidence scores of items for all selected classifications and classifying the 
items based on confidence score comparison (col. 2, lines 36-42). 

In regards to dependent claim 6, Stubler discloses wherein the multimedia 
content comprises one or more types selected from the group consisting otaudio, 
video, Web pages, time series data, surveillance data, sensor data, and XML data (col. 
3 line 46 - col. 4 line 12; Stubler discloses images type multimedia. Using the broadest 
reasonable interpretation, the Examiner concludes surveillance data to include but not 
be limited to images.). 

In regards to dependent claim 10, Stubler discloses the method of claim 1, 
wherein the input annotations are created by means of steps selected from the group 
consisting of: creating new annotations, deleting existing annotations, rejecting 
proposed annotations, and modifying annotations (col. 9, line 65-col. 10, line 18; Stubler 
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disclose an interactive user verification stage in which the user may select and/or edit 
captions and label.). 

In regards to dependent claim 23, Stubler in view of LeBrun does not expressly 
disclose wherein the at least one criterion includes an ambiguity level of the selected 
examples. 

However Neal teaches wherein the at least one criterion includes an ambiguity 
level of the selected examples (col. 1 1 , lines 1-47; Neal teaches a classification 
confidence score which determines the level of confidence in which a category is likely 
to be correct during classification. If the item has a high confidence, then it can be 
classified directly. If the confidence level is low, then the results can be sent to the user 
interface for review and selection by the operator. Using the broadest reasonable 
interpretation, the Examiner has determined that the "confidence level" as taught by 
Neal is analogous with the "ambiguity level" of the current invention.). 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Stubler and LeBrun with Neal for the benefit of 
comparing confidence scores of items for all selected classifications and classifying the 
items based on confidence score comparison (col. 2, lines 36-42). 

Note 

It is noted that any citations to specific, pages, columns, lines, or figures in the 
prior art references and any interpretation of the reference should not be considered to 
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be limiting in any way. A reference is relevant for all it contains and may be relied upon 
for all that it would have reasonably suggested to one having ordinary skill in the art. 
See MPEP 2123. 

Claims 2, and 5 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Stubler and LeBrun in view of Neal, further in view of Lennon et al. (Patent 
No.: 6,718,063 B1; Filed Dec. 10, 1999) (hereinafter "Lennon"). 

In regards to dependent claim 2, Stubler discloses wherein the step of actively 
selecting is performed using a selection technique selected from deterministic (col. 4 
line 64 -col. 5 line 19). 

Stubler and LeBrun in view of Neal does not expressly disclose actively selecting 
is performed using a selection technique of probabilistic. 

However Lennon teaches actively selecting is performed using a selection 
technique of probabilistic (col. 3, lines 19-35; Lennon teaches using a probabilistic 
method for correctly assigning labels to regions of images.). 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Lennon with Stubler and LeBrun in view of Neal for 
the benefit of using a probabilistic method, which results in each assigned label for a 
region having an associated probability or likelihood of the label being correctly 
assigned (col. 3, lines 23-27). 
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In regards to dependent claim 5, Stubler and LeBrun in view of Neal does not 
expressly disclose wherein an optimization criterion for active selection includes one or 
more criteria selected from the group consisting of: information measures. 

However Lennon teaches wherein an optimization criterion for active selection 
includes one or more criteria selected from the group consisting of: information 
measures (col. 3, lines 34-36; col. 3, line 63-col. 4, line 4; Lennon teaches comparing 
the distance metric with a predetermined threshold in order to determine the similarity of 
the images. Thus Lennon teaches active selection includes one or more criteria 
selected from the group consisting of: information measures, using distance and 
threshold measures.). 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Lennon with Stubler and LeBrun in view of Neal for 
the benefit of using a probabilistic method, which results in each assigned label for a 
region having an associated probability or likelihood of the label being correctly 
assigned (col. 3, lines 23-27). 

Note 

It is noted that any citations to specific, pages, columns, lines, or figures in the 
prior art references and any interpretation of the reference should not be considered to 
be limiting in any way. A reference is relevant for all it contains and may be relied upon 
for all that it would have reasonably suggested to one having ordinary skill in the art. 
See MPEP 2123. 
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Claims 7, 8, 11, 13-15 and 29-31 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Stubler and LeBrun in view of Neal, further in view of Lipson et 
al. (Patent No.: 5,963,670; Filed: Feb. 12, 1996) (hereinafter "Lipson"). 

In regards to dependent claim 7, Stubler and LeBrun in view of Neal does not 
expressly disclose the method of claim I, wherein the input annotations are created by a 
user with reference to a vocabulary. 

However Lipson teaches the method of claim I, wherein the input annotations are 
created by a user with reference to a vocabulary (col. 9, lines 23-35; Lipson teaches a 
rich vocabulary to differentiate between many classes of images.). 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Lipson with Stubler and LeBrun in view of Neal for the 
benefit of providing a class model, which can be used to detect images of that class in a 
database (col. 2, lines 39-41). 

In regards to dependent claim 8, Stubler and LeBrun in view of Neal does not 
expressly disclose the method of claim 7, wherein the vocabulary contains one or more 
items selected from the group consisting of: terms, concepts, labels, and annotations. 

However Lipson teaches the method of claim 7, wherein the vocabulary contains 
one or more items selected from the group consisting of: terms, concepts, labels, and 
annotations (col. 9, lines 23-42; Lipson teaches a rich vocabulary to differentiate 
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between many classes of images. It has been established and is commonly known that 
vocabularies typically contain terms and concepts.). 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Lipson with Stubler and LeBrun in view of Neal for the 
benefit of providing a class model, which can be used to detect images of that class in a 
database (col. 2, lines 39-41). 

In regards to dependent claim 11, Stubler and LeBrun in view of Neal does not 
expressly disclose the method of claim 7, wherein the vocabulary is adaptively or 
dynamically organized and/or limited by the system or the user. 

However Lipson teaches the method of claim 7, wherein the vocabulary is 
adaptively or dynamically organized and/or limited by the system or the user (col. 9, 
lines 23-35.). 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Lipson with Stubler and LeBrun in view of Neal for the 
benefit of providing a class model, which can be used to detect images of that class in a 
database (col. 2, lines 39-41). 

In regards to dependent claim 13, Stubler discloses the method of claim 1, 
wherein the determination of the propagation of annotations is made deterministically or 
probabilistically (col. 4 line 64 - col. 5 line 19) 
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Stubler and LeBrun in view of Neal does not expressly disclose the use of 
models for each annotation or for joint annotations. 

However Lipson teaches the use of models for each annotation or for joint 
annotations (col. 2, lines 56-59; Lipson teaches an image model for classifying or 
detecting images.). 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Lipson with Stubler and LeBrun in view of Neal for the 
benefit of providing a class model, which can be used to detect images of that class in a 
database (col. 2, lines 39-41). 

In regards to dependent claim 14, Stubler and LeBrun in view of Neal does not 
expressly disclose the method of claim 13, wherein the models are created or learned 
automatically or semi-automatically and/or are updated adaptively from interaction with 
the user. 

However Lipson teaches wherein the models are created or learned 
automatically or semi-automatically and/or are updated adaptively from interaction with 
the user (col. 9, line 37-col. 10, line 22; Lipson teaches a method for generating a class 
model.). 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Lipson with Stubler and LeBrun in view of Neal for the 
benefit of providing a class model, which can be used to detect images of that class in a 
database (col. 2, lines 39-41). 
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In regards to dependent claim 15, Stubler and LeBrun in view of Neal does not 
expressly disclose the method of claim 13, wherein the models are based on nearest 
neighbor voting or variants, parametric or statistical models, expert systems, rule-based 
systems, or hybrid techniques. 

However Lipson teaches wherein the models are based on nearest neighbor 
voting or variants (col. 2, lines 56-59; col. 13, line 2-col. 14, Iine18). 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Lipson with Stubler and LeBrun in view of Neal for the 
benefit of providing a class model, which can be used to detect images of that class in a 
database (col. 2, lines 39-41). 

In regards to dependent claim 29, Stubler and LeBrun in view of Neal does not 
expressly disclose the method of claim 13, wherein the models are based on expert 
systems. 

However Lipson teaches wherein the models are based on expert systems (col. 
8, lines 53-60; Lipson teaches a model in which a user has personal knowledge of 
relevant relative relationship for a given class of images. Thus Lipson teaches wherein 
the models are based on expert systems.). 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Lipson with Stubler and LeBrun in view of Neal for the 



Application/Control Number: 10/056,546 Page 14 

Art Unit: 2176 

benefit of providing a class model, which can be used to detect images of that class in a 
database (col. 2, lines 39-41). 

In regards to dependent claim 30, Stubler and LeBrun in view of Neal does not 
expressly disclose the method of claim 13, wherein the models are based on rule-based 
systems. 

However Lipson teaches wherein the models are based on rule-based systems 
(col. 3, lines 29-35; col. 17, lines 10-42; Lipson teaches deformable image template 
models. Is has been established and is well known in the art that template are rule- 
based.). 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Lipson with Stubler and LeBrun in view of Neal for the 
benefit of providing a class model, which can be used to detect images of that class in a 
database (col. 2, lines 39-41). 

In regards to dependent claim 31, Stubler and LeBrun in view of Neal does not 
expressly disclose the method of claim 13, wherein the models are based on hybrid 
techniques. 

However Lipson teaches wherein the models are based on hybrid techniques 
(col. 2, lines 42-54; col. 13, line 56-col. 14, line 18; Lipson teaches a model which is 
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provided from spatial and photometric properties of an image. Thus Lipson teaches 
models are based on hybrid techniques, a combination of models/relationships.). 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Lipson with Stubler and LeBrun in view of Neal for the 
benefit of providing a class model, which can be used to detect images of that class in a 
database (col. 2, lines 39-41). 

Note 

It is noted that any citations to specific, pages, columns, lines, or figures in the 
prior art references and any interpretation of the reference should not be considered to 
be limiting in any way. A reference is relevant for all it contains and may be relied upon 
for all that it would have reasonably suggested to one having ordinary skill in the art. 
See MPEP 2123. 
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Claims 9 and 12 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Stubler and LeBrun in view of Neal, further in view of Chino et al. (Patent 
No.: 6,118,888; Filed: Feb. 25, 1998) (hereinafter "Chino"). 

In regards to dependent claim 9, Stubler and LeBrun in view of Neal does not 
expressly disclose the method of claim 1, wherein the process of creating input 
annotations by the user involves multimodal interaction with the user. 

However Chino teaches wherein the process of creating input annotations by the 
user involves multimodal interaction with the user (col. 3, lines 17-65; col. 30, lines 13- 
15; Chino teaches creating input annotations by the user involves multimodal interaction 
with the user.). 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Chino with Stubler and LeBrun in view of Neal for the 
benefit of providing a multimodal interface apparatus and method to smoothly 
communicate between the user and the apparatus using the user's gaze object (col. 3, 
lines 17-20). 

In regards to dependent claim 12, Stubler and LeBrun in view of Neal does not 
expressly disclose the method of claim 9, wherein the multimodal interaction involves 
one or more elements selected from the group consisting of: gaze detection, finger 
pointing, expression detection, and effective computing methods for sensing a user's 
state. 
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However Chino teaches wherein the multimodal interaction involves one or more 
elements selected from the group consisting of: gaze detection, finger pointing, 
expression detection, and effective computing methods for sensing a user's state 
(Abstract; col. 17, lines 14-58; Chino teaches a multimodal interface which obtains 
position information of the objected pointed to by the gesture input, the user's face or 
hand moved by the gesture input.). 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Chino with Stubler and LeBrun in view of Neal for the 
benefit of providing a multimodal interface apparatus and method to smoothly 
communicate between the user and the apparatus using the user's gaze object (col. 3, 
lines 17-20). 

Note 

It is noted that any citations to specific, pages, columns, lines, or figures in the 
prior art references and any interpretation of the reference should not be considered to 
be limiting in any way. A reference is relevant for all it contains and may be relied upon 
for all that it would have reasonably suggested to one having ordinary skill in the art. 
See MPEP 2123. 
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Claim 24 is rejected under 35 U.S.C. 103(a) as being unpatentable over 
Stubler and LeBrun in view of Neal, further in view of Toyama (Patent No.: US 
6,816,847 B1; Filling Date: Sep. 23, 1999). 

In regards to dependent claim 24, Stubler in view of LeBrun does not expressly 
disclose wherein the at least one criterion includes a confidence level of the selected 
examples, the confidence level being inversely proportional to a distance of a new 
feature of the selected examples from a separating hyperplane in an induced higher 
dimensional feature space. 

Neal teaches a confidence level of the selected examples, the confidence level 
being inversely proportional to a distance of a new feature of the selected examples 
(col. 1 1 , lines 1-47; Neal teaches a classification confidence score which determines the 
level of confidence in which a category is likely to be correct during classification. If the 
item has a high confidence, then it can be classified directly. If the confidence level is 
low the results can be sent to the user interface for review and selection by the 
operator. Using the broadest reasonable interpretation, the Examiner has determined 
that the "confidence level" as taught by Neal is analogous with the "ambiguity level" of 
the current invention. Thus a high confidence score is inversely proportional to distance 
of the new feature, which determines whether or not to automatically classify the 
item/category). 
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Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Stubler and LeBrun with Neal for the benefit of 
comparing confidence scores of items for all selected classifications and classifying the 
items based on confidence score comparison (col. 2, lines 36-42). 

Toyama teaches a distance of a new feature of the selected examples from a 
separating hyperplane in an induced higher dimensional feature space (col. 5, line 47- 
col. 6, line 50; Toyama teaches a SVM classifier by identifying a hyperplane that 
separates a set of positive and negatives examples with a maximum margin. As 
Applicant disclosed within the specification (0008), for SVM classifiers the distance of 
an unlabeled data-point from the separating hyperplane in the high dimensional feature 
space could be taken as a measure of uncertainty (alternatively, a measure of 
confidence in classification) of the data-point. Therefore Toyama teaches a distance of 
a new feature of the selected examples from a separating hyperplane in an induced 
higher dimensional feature space. Toyama further teaches the quality of the learned 
classifiers for aesthetic image judgment can be enhanced by inputting to the feature 
selection procedures that are useful for distinguishing different aesthetic among images. 
Using the broadest reasonable interpretation, the Examiner has determined that the 
"aesthetic score" as taught by Neal is analogous with the "confidence level" of the 
current invention. 

Using the broadest interpretation, the Examiner concludes at the time of the 
invention, one of ordinary skill of the art could modify Neal's teaching with Toyama 



Application/Control Number: 10/056,546 Page 20 

Art Unit: 2176 

teaching of enhancing the learned qualifier in such a way that wherein the at least one 
criterion includes a confidence level of the selected examples, the confidence level 
being inversely proportional to a distance of a new feature of the selected examples 
from a separating hyperplane in an induced higher dimensional feature space.). 

Therefore at the time of the invention it would have been obvious to one of 
ordinary skill in the art to combine Toyama with Stubler, LeBrun and Neal for the benefit 
of generating a set of images for input into a SVM classifier (col. 1 , lines 57-64). 

Note 

It is noted that any citations to specific, pages, columns, lines, or figures in the 
prior art references and any interpretation of the reference should not be considered to 
be limiting in any way. A reference is relevant for all it contains and may be relied upon 
for all that it would have reasonably suggested to one having ordinary skill in the art. 
See MPEP 2123. 

Response to Arguments 

Applicant's arguments have been considered but are moot in view of the new 
ground(s) of rejection. A new ground(s) of rejection is made in view of Stubler, LeBrun, 
Lennon, Lipson, Neal, Toyama and Chino. 
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Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to James J. Debrow whose telephone number is 571-272- 
5768. The examiner can normally be reached on 8:00-5:00pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Doug Hutton can be reached on 571-272-4137. The fax phone number for 
the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 

JAMES DEBROW 
EXAMINER 
ART UNIT 2176 

I (Doug Hutton/ 
Doug Hutton 
Supervisory Primary Examiner 
Technology Center 2100 



